Skip to content

csnode: backfill signatures verified on worker threads - #152

Merged
EnzinoBB merged 4 commits into
masterfrom
feat/backfill-parallel-verify
Nov 2, 2026
Merged

EnzinoBB merged 4 commits into
masterfrom
feat/backfill-parallel-verify

Conversation

@EnzinoBB

@EnzinoBB EnzinoBB commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

Why

Measured on the mainnet test node (master b703913, 1.4 M blocks), per backfilled block on the processor thread: verify (signatures plus sequence/hash checks) ≈ 1000 µs, confirmations (the round confirmations of the block stored before against this block's confidants) ≈ 1000 µs, store ≈ 500 µs. The backfill held the processor thread 97–98 % of the time (turn budgets of #140/#146 plus the drains in onBlocks). Moving the two signature checks to worker threads should leave about 0.5 ms a block on the processor thread.

Design

  • BackwardSync::Settings::verifyThreads (config [storage] backfill_verify_threads, 0..16, default min(4, cores − 1), at least 1; 0 = everything on the processor thread as before). The threads start lazily at the first job and are joined by stop() and the destructor, like the flush thread of csnode: the periodic backfill flush runs on its own thread #148.
  • Signatures: when a block enters buffer_ (onBlocks), its hash is computed (and cached in the shared csdb::Pool data) on the processor thread, and a job with a copy of the pool plus a shared_ptr<SignatureCheck> (atomic<bool> ready + why) is queued. The queue is ordered highest sequence first, since the storage always waits for the highest buffered block.
  • drain() keeps the sequence and hash-chain checks (cheap) on the processor thread, in the same order as before (sequence, hash, then signatures). If the block's check is ready its result is used — a failure goes through reject() exactly as today; if not, the turn ends (no waiting) and the next tick/reply goes on.
  • Stale results: the SignatureCheck is owned by the Buffered entry. A block that is rejected, replaced by a re-sent copy, or dropped at finish() leaves its result to nobody; the verifier skips a job whose check nobody holds any more (use_count() == 1). No keying by sequence/hash is needed.
  • Round confirmations (deviation from the suggested pairing at onBlocks): they are queued right after the block is stored, as a job with copies of lastStored_ (the block above) and the stored block, i.e. exactly the pairs checked today. They decide nothing (a mismatch is counted and logged, never a rejection), so nothing needs to wait for them. Pairing at buffer time would have needed results keyed by the upper block's hash, plus a fallback when the upper block changed. One consequence: confirmationErrors and its log line can come slightly after the block is stored, and a stop() drops the checks still queued. After the end of the backfill, the threads finish the queued confirmations and return.
  • Timing (csnode: backfill progress logs the average cost of a block per phase #151): phaseVerifyUs_/phaseConfirmUs_ are atomics that the verifier threads add to. busyUs / "processor thread busy %" count only processor-thread time, including the hashing done in onBlocks. The progress line says when verify and confirmations ran on the verifier threads. The slow-turn log still reports processor time only.
  • onBlocks after stop() now drops the reply (the verifiers are gone, and the node is closing its storage).
  • If no thread can be started (std::system_error), it logs and falls back to verifyThreads = 0.

Unchanged: the verification predicates (BlockVerifier::verifySignatures / verifyConfirmations), the order of storing, block 0 (genesis without signatures: verifySignatures accepts sequence 0), floor/finish, retries and peer marking.

Thread safety

  • A verifier only ever touches its own csdb::Pool copies, the job's SignatureCheck, the atomic counters (confirmationErrors_, phaseVerifyUs_, phaseConfirmUs_), and log() / hooks_.onLog / hooks_.onVerify, which are set before any thread starts and only read afterwards. It never reads buffer_, next_, lastStored_ or any other processor-thread state.
  • csdb::Pool is a ref-counted copy-on-write handle with an atomic count. Its const getters used by BlockVerifier are reads, except hash(), which computes the hash lazily through a non-const access (detach). The hash is therefore computed on the processor thread before the copy is shared, so no thread writes the shared data. A later non-const access on either side detaches (copies) while the count is > 1, which is a read of the shared data. BlockChain::isBootstrap, NodeUtils::checkGroupSignature and cscrypto/libsodium verification are pure.
  • why is written before ready.store(release) and read after ready.load(acquire).
  • stop(): it sets verifyStopping_ under verifyMutex_, takes the queue out (the queued jobs are dropped), notifies, and joins outside the lock. A verifier in the middle of a check holds no lock, so it ends that check and returns.
  • Locally (outside CI), the backward-sync tests were built against this branch's sources with -fsanitize=thread: 27/27 pass with no ThreadSanitizer report, also 30 repetitions of the threaded tests pinned to one core.

Tests

  • settingsOf() sets verifyThreads = 0, so all existing tests run on the processor thread as before. TearDown() destroys the sync while the fixture it logs to still exists. The shared Gate/Release of the flush test moved to the namespace so the new tests can reuse it.
  • VerifierThreadsStoreTheSameBlocks (3 threads): out-of-order replies are stored strictly downwards to block 0; the block 350 confirmation mismatch is counted exactly once.
  • VerifierThreadsRefuseBadSignatures (2 threads): a bad signature on block 400 is rejected, the peer is marked, the batch is re-requested from another peer, and the backfill completes.
  • ATurnDoesNotWaitForTheVerifiers: with the verifiers held at a gate, onBlocks and ticks return without storing anything (100 buffered); once the gate opens the blocks are stored.
  • StopEndsTheVerifiers: stop() while a verifier is mid-check returns once that check ends, and nothing more is stored.
  • BackfillConfig.VerifyThreads: the default (unset), 0, 3, clamped to 16, and bad values leave it unset.
  • The threaded tests wait with bounded real-time loops (60 s) that move the simulated clock 1 ms per tick, so no batch times out on a slow runner.

For the reviewer

  • With the signatures off the processor thread, backfill_store_per_turn (default 50) becomes the cap at 10 ticks/s plus the drains in replies: about 500+ blocks/s. It may be worth raising after a mainnet measurement.
  • The confirmation-error counter can lag the stored count (see above).

🤖 Generated with Claude Code

EnzinoBB and others added 4 commits November 2, 2026 07:20
The backfill held the processor thread 97-98 % of the time on mainnet,
about 2.5 ms a block: 1 ms for the block's signatures, 1 ms for the
round confirmations of the block stored before, 0.5 ms to store it.

A pool of verifier threads (Settings::verifyThreads, [storage]
backfill_verify_threads, 0..16, default min(4, cores - 1); 0 keeps
everything on the processor thread) now checks the signatures of each
block as it enters the buffer, highest blocks first, on a copy of the
block whose hash was computed before it was shared. drain() keeps the
sequence and hash-chain checks, takes the result when it is ready and
ends the turn otherwise, never waiting. The result lives in a
shared_ptr owned by the buffer entry: a block refused, replaced or
dropped leaves its result to nobody, and the verifier skips a check
nobody waits for. The round confirmations of the block stored before
are checked on the verifier threads after the store, with the same
predicate; a mismatch is counted and logged as before, never refused.

stop() and the destructor join the threads (a check under way ends
first, the queued ones are dropped); after the end of the backfill the
threads finish the queued confirmations and return. The per-phase
averages of #151 add the verifier threads' time atomically; busyUs and
the busy share count only the processor thread.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- signature checks are served before the confirmation checks (only the
  former gate the storage), each highest block first
- a failure to grow the thread vector falls back like a thread that
  cannot start (reserve, catch std::exception)
- the slow-turn line no longer reports a confirmations time when the
  signatures run on the verifier threads
- test: wait for the confirmation line itself, not only the counter
  that goes up before it is logged

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Confirmation jobs are served after the signature jobs; on a single slow
verifier they could pile up, each holding two blocks. Past 2000 queued,
the processor thread checks the confirmations of the next stored blocks
itself (the existing fallback when submit() refuses), which also slows
the storing down until the verifiers catch up.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@EnzinoBB
EnzinoBB merged commit 1b6438d into master Nov 2, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant